Neural networks are a class of electronic circuits which emulate higher-order brain function such as memory, learning and/or perception/recognition. Associative networks are one category of neural devices which sense an input event and output a pattern of signals identifying that event.
Associative networks generally take the form of a matrix comprising a set of horizontal lines which cross and contact a set of vertical lines. The horizontal lines simulate the function of axons in the cortex of the brain and are used as inputs. The vertical lines simulate the function of dendrites extending from neurons. Each vertical line terminates at a voltage summing device which acts to simulate the function of the neuron cell body. Examples of such associative networks are found in co-pending applications entitled; "Semiconductor Cell For Neural Network Employing A Four-Quadrant Multiplier", Ser. No. 283,553, filed Dec. 9, 1988; "EXCLUSIVE-OR Cell For Neural Network And The Like", Ser. No. 309,247, filed Feb. 10, 1989; and "Neural Network Employing Leveled Summing Scheme With Blocked Array", Ser. No. 357,411, filed May 26, 1989, all of which are assigned to the assignee of the present application.
Within an associative network, neural synapses are simulated by circuit cells which provide electrical connection between the horizontal and vertical lines of the network. Individual synapses provide a weighted electrical connection between an input and a voltage summing element, i.e., a neuron. Basically, the synapse cell outputs the product of an input signal and a weight value stored within the synapse. These synapse cells may either be analog or digital in nature. Analog circuitry is often preferred over digital circuitry for neural networks because of its superior density.
For an analog implementation, the weighted sum of input signals is usually computed by summing analog currents or charge packets. Examples of circuit devices useful as synapse cells in neural networks are described in the co-pending applications entitled, "Adaptive Synapse Cell Providing Both Excitatory And Inhibitory Connections In An Associative Network", Ser. No. 379,933, filed Jul. 13, 1989; and "Improved Synapse Cell Employing Dual Gate Transistor Structure", Ser. No. 419,685, filed Oct. 11, 1989, which applications are also assigned to the assignee of the present invention.
In these exemplary devices, the synapse cells employ floating gate transistors which store charge representing a weighted value of the synapse connection (i.e., the connection strength). Floating gate transistors take a variety of forms in the prior art, some being both electrically erasable and electrically programmable and others requiring, for example, ultraviolet light for erasing. Most often, the floating gate of these devices is made up of a polycrystalline silicon (i.e., polysilicon) member which is completely surrounded by an insulative layer. These non-volatile storage devices are frequently referred to in the prior art as EPROMs, EEPROMs or flash EPROMs. The currently preferred embodiment of the present invention employs standard EEPROMs, however, ordinary EPROMs or flash EPROM devices may also be utilized with similar results.
One of the major limitations associated with analog neural networks that employ floating gate devices is the inability of the synapse cells in the neural network to maintain high accuracy for an extended period of time. Accuracy is defined as the absolute value, measured in either voltage or charge, to which the synapse weight is set. As time passes, the programmed weight values stored in the synapse cells tend to dwindle due to charge redistribution which occurs in the dielectric material surrounding the floating gate member. These changes can result in erroneous products of the input signals and weights, thereby lowering the overall accuracy of the neural network.
To be more specific, the dielectric (e.g., silicon dioxide) which covers and insulates the polysilicon floating gate from the control gate of the EEPROM, relaxes with time. This relaxation redistributes the charge already present within the dielectric material. Effectively, this charge redistribution causes a shift in the threshold voltage, V.sub.th, of the floating gate transistor. The threshold voltage V.sub.th is one measure of the weight stored within a synapse cell. Hence, any shift in charge distribution translates into a corresponding V.sub.th shift.
Such non-programmed and unintentional decreases in V.sub.th lead to inaccuracies in the sum-of-products (SOP) computed by summing the products of stored weights and input values at the output of the network. Ideally, each synapse cell should be able to store its set weight value throughout the life of the neural network. However, because of the dielectric relaxation phenomenon, the stored weight programmed onto the floating gate diminishes over time. Moreover, since the charge redistribution occurs gradually, the cell's weight value varies continuously over time with a decreasing rate. Thus, the overall, long-term accuracy of the neural network is seriously affected by this charge redistribution phenomenon. It is appreciated by practitioners in the art that without high accuracy, the reliability of a neural network is significantly hampered or its information capacity is seriously limited.
The need for high accuracy exists not only in neural networks which rely upon floating gate technology, but also in those networks which employ volatile memories as well. In fact, volatile memory neural networks face even greater difficulties in achieving high accuracy. By way of example, in a neural network utilizing dynamic memory cells, dynamic storage of the weight must be performed--typically using a capacitive device such as a dynamic random-access memory (DRAM) cell. The drawback with using DRAMs is that charge is constantly leaking from the memory cell. This discharge mandates that the DRAM be continuously refreshed. Consequently, a separate memory chip containing digitally stored values must be provided. If the weights are stored externally, the increase in chip area resulting from the additional DRAMs will dramatically increase the manufacturing cost of the entire neural network system.
Alternatively, an on-chip refresh circuit may be implemented that captures the volatile analog levels of the memory and compares them to a finite set of quantized levels that are equally spaced. Refresh is performed more often than the time it takes the fastest leaking cell to decay by the voltage difference between adjacent quantized levels. At the time of refresh the analog level is reset to the next higher quantized level.
However, this approach also suffers from serious drawbacks. Foremost among these is the fact that the accuracy of the stored analog weights is always limited by the difference between the quantized levels. The quantized levels cannot be spaced too closely as this will require more frequent refresh and may lead to errors in comparing the analog levels to the quantized levels. Faster refresh is required when the quantized levels are spaced more closely because it takes the analog level less time to decay past the next lower quantized level. If an analog level decays past the next lower quantized level before it is refreshed, it will be refreshed to a value equal to that next lower level and its original value will be lost. In the past, this has been a fundamental limitation for these circuits.
Therefore, what is needed is a new method for increasing the accuracy of analog synaptic connections. A method which compensates for the redistribution of charge which naturally and predicatably occurs in a floating gate memory device is one such solution. As will be seen, the present invention greatly increases the accuracy of the weights by performing bake and retraining cycles which compensate for any charge redistribution that occurs.